Papers with teacher network
Do we need Label Regularization to Fine-tune Pre-trained Language Models? (2023.eacl-main)
Copied to clipboard
Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh, Tianda Li, Alan Do-Omri, Peng Lu, Pascal Poupart, Ali Ghodsi
| Challenge: | Knowledge Distillation (KD) is a label regularization technique that can be replaced with lighter teacher-free variants such as the label-smoothing technique. |
| Approach: | They propose to use knowledge distillation to train student models by deploying the teacher network during training. |
| Outcome: | The proposed method can be replaced with lighter teacher-free variants on PLMs with more than 600 distinct trials and ran each configuration five times. |
Contrastive Distillation on Intermediate Representations for Language Model Compression (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to compress language models use a simple L_2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one. |
| Approach: | They propose a method that uses knowledge distillation to distill knowledge through intermediate layers of the teacher via a contrastive objective. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark. |
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques are not suitable for deep learning tasks due to memory constraints. |
| Approach: | They propose to combine knowledge from a large teacher network into a student network (S) they propose to use a combinatorial mechanism to inject layer-level supervision from T to S . |
| Outcome: | The proposed model outperforms existing models in PortugueseEnglish, TurkishEnglish and EnglishGerman directions and students trained using it have 50% fewer parameters and can deliver comparable results to 12-layer teachers. |
Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for training language-vision models only consider monolingual learning, especially English. |
| Approach: | They propose to extend an English language-vision model into a multilingual and code-mixed model by using knowledge distillation techniques. |
| Outcome: | The proposed model outperforms existing models on multilingual and code-mixed VQA datasets on eleven languages. |
Semantic Alignment with Calibrated Similarity for Multilingual Sentence Embedding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for learning semantic similarity between two English sentences have focused on one sub-task and therefore showed biased performance. |
| Approach: | They propose a method to learn semantic similarity between two English sentences using siamese networks. |
| Outcome: | The proposed method improves on both sub-tasks and predicts similarity scores in 14 languages. |
BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover’s Distance (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have been proposed and applied to many NLP tasks, yielding state-of-the-art performance, but high storage and computational costs obstruct them to be effectively deployed on resource-constrained devices and real-time applications. |
| Approach: | They propose a BERT distillation method which allows each intermediate student layer to learn from any intermediate teacher layers. |
| Outcome: | The proposed method can learn from different teacher layers adaptively for different NLP tasks. |
Amalgamating Knowledge from Two Teachers for Task-oriented Dialogue System with Adversarial Training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to task-oriented dialogue systems require a large number of handcrafted features and labels. |
| Approach: | They propose a "Two-Teacher One-Student" learning framework for task-oriented dialogue . the framework amalgamates knowledge from two teacher networks and provides guidance . |
| Outcome: | The proposed framework outperforms baseline methods on two benchmark datasets . it can retrieve accurate KB entities and generate human-like responses simultaneously . |
Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) methods require a substantial quantity of high-quality annotation for training models. |
| Approach: | They propose a method to reduce the number of incorrect pseudo labels in self-training . they propose 'uncertainty-aware teacher learning' and 'student-student collaboration' |
| Outcome: | The proposed method is superior to state-of-the-art DS-NER denoising methods. |
Leveraging Training Dynamics and Self-Training for Text Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Semi-supervised learning (SSL) is a promising technique for improving deep learning models when training data is scarce. |
| Approach: | They propose a semi-supervised learning approach that leverages training dynamics of unlabeled data. |
| Outcome: | The proposed method achieves an average increase in F1 score of 3.5% over baselines in low resource settings. |
On Knowledge distillation from complex networks for response prediction (N19-1)
Copied to clipboard
| Challenge: | Recent advances in Question Answering have led to the development of very complex models . however, these models are expensive in space and time and require limited resources . |
| Approach: | They propose to use simple models which learn to emulate characteristics of a teacher network . they use a 12GB Tesla K80 GPU to restrict the maximum length of the input document . |
| Outcome: | The proposed model can perform better on a Holl-E dialog dataset. |
Text Simplification via Adaptive Teaching (2024.findings-acl)
Copied to clipboard
| Challenge: | Text simplification is the process of rewriting a text using simpler vocabulary and grammatical structure in order to make it more accessible and understandable for a larger audience. |
| Approach: | They propose a model for text simplification based on adaptive teaching using a teacher network and a text generation network. |
| Outcome: | The proposed model outperforms the current state-of-the-art model on the Wiki-Doc and D-Wikipedia datasets and performs well on human evaluations in terms of text simplicity, correctness, and fluency. |
Patient Knowledge Distillation for BERT Model Compression (D19-1)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT have proven to be highly effective for natural language processing tasks, but the high demand for computing resources hinders their application in practice. |
| Approach: | They propose to compress an original large model (teacher) into an equally-effective lightweight shallow network (student) Empirically, this translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy. |
| Outcome: | The proposed model reduces the computational cost of training models using the teacher model into a lightweight shallow network. |
BERT Learns to Teach: Knowledge Distillation with Meta Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods are based on teacher model, but have drawbacks . a teacher model is fixed during training, but meta learning can improve student performance . |
| Approach: | They propose a meta learning framework that allows the teacher network to learn to better transfer knowledge to the student network. |
| Outcome: | Experiments show that MetaDistil can improve on existing methods and is less sensitive to student capacity and hyperparameters. |
Training a Better Chinese Spelling Correction Model via Prior-knowledge Guided Teacher (2024.findings-acl)
Copied to clipboard
| Challenge: | Chinese Spelling Correction models are prone to over-correct and poor generalization for error patterns outside the standard distribution. |
| Approach: | They propose a teacher network guided by prior knowledge for distillation learning of CSC models. |
| Outcome: | The proposed method significantly enhances the CSC model’s language modeling capabilities, crucial for minimizing over-correction. |
Learn From One Specialized Sub-Teacher: One-to-One Mapping for Feature-Based Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical results show that our proposed approach outperforms the state-of-the-art methods by maintaining higher performance on most benchmark datasets. |
| Approach: | They propose to break down the global feature distillation task into N local sub-tasks and make each focused sub-student learn from one specialized sub-teacher. |
| Outcome: | The proposed method outperforms state-of-the-art methods on most benchmark datasets while maintaining higher performance. |